Face Detection and Lip Localization

Integration of audio and video signals for automatic speech recognition has become an important field of study. The AudioVisual Speech Recognition (AVSR) system is known to have accuracy higher than audio-only or visual-only system. The research focused on the visual front end and has been centered around lip segmentation. Experiments performed for lip feature extraction were mainly done in constrained environment with controlled background noise. In this thesis we focus our attention to a database collected in the environment of a moving car which hampered the quality of the imagery. We first introduce the concept of illumination compensation, where we try to reduce the dependency of light from over-or underexposed images. As a precursor to lip segmentation, we focus on a robust face detection technique which reaches an accuracy of 95%. We have detailed and compared three different face detection techniques and found a successful way of concatenating them in order to increase the overall accuracy. One of the detection techniques used was the object detection algorithm proposed by Viola-Jones. We have experimented with different color spaces using the Viola-Jones algorithm and have reached interesting conclusions. Following face detection we implement a lip localization algorithm based on the vertical gradients of hybrid equations of color. Despite the challenging background and image quality, success rate of 88% was achieved for lip segmentation. v ACKNOWLEDGMENTS The end of this thesis marks the culmination of my academic career at Cal Poly. My experiences at Cal Poly have inspired and shaped me to be thoroughly involved in my subject and in love with what I do. I dedicate this thesis to my family, my mom Tehmina and my brother Talib. Both of you have supported me in every way possible. I also want to thank Chiweng Kam, without whose guidance, motivation and positive attitude it would have been impossible to have seen this thesis through. I must also express my gratitude for Dr. Xiaozheng (Jane) Zhang for her guidance throughout this endeavor, her review of my work, and for the freedom in direction she allowed me to take on this project. I must also thank Dr John Saghri and Dr Fred DePiero for serving on my thesis committee and for fostering my interest in image processing and pattern recognition.